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ABSTRACT 

Evaluations of the Botswana National Literacy Program 
(BNLP) are reviewed, with emphasis on one internal and one external 
evaluation. The BNLP may be one of the most extensively evaluated 
national literacy programs ever conducted, with a program staff 
intensively trained in evaluation. One lesson of the repeated 
evaluations of the BNLP is that a point comes when evaluation may use 
resources needed for program implementation. Evaluation cannot be 
fully exorcised of politics; however, the political ramifications 
must at least be clarified and the worst political encroachments 
resisted. Internal evaluations seemed to serve program implementation 
better and demonstrated the usefulness of the Action Training Model. 
The need to intervene and control partly explained when and why the 
various evaluations were commissioned and what use was made of 
evaluation results. (SLD) 
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EVALUATIONS OF BOTSWANA NATIONAL LITERACY PROGRAM: 
THE POLITICS OF RENOVATION AND CONTROL 



To evaluate is to judge the merit and worth of what is 
being evaluated (Cuba and Lincoln, 1981). But judgment must be 
rendered within a contexture of competing values. As a result, 
the professional commitment to renovate a program gets confounded 
with the political commitment to control, and, thereby, to obtain 
a particular distribution of social goods in a particular social 
setting (Bhola, 1975). Evaluation becomes a political act 
(House, 1973; Palumbo, 1987; Rossi and Williams, 1972). 

The professional and the political in evaluation intersect 
at many points of the evaluation process: in the initial decision 
to evaluate a program, in the choice of tne time-frame of 
evaluation, in the selection of evaluators and of samples and 
methods, in the establishment of norms of success or failure, 
and, of course, in the utilization of results (Glover and 
Strawbridge, 1985; Hall, 1978; Lindblom and Cohen, 1979; Weiss, 
1976) . 

An important issue in program evaluation is internal 
evaluation versus external evaluation. Internal evaluation is 
conducted within the institution that is also responsible for the 
program, ideally, by the same people who are responsible for the 
implementation of the program being evaluated. External 
evaluation is conducted by agents external to th^ institution 
responsible for the program, and sometimes external to the 
culture. The debate on the relative merits and demerits of 
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internal and external evaluation has been conducted in scientific 
terms of objectivity of results. It does not seem to be realized 
that neither internal nor external evaluations per se have a 
monopoly on objectivity or on meaningf ulness ; that objectivity 
may not be the point; and that the point may indeed be the locus 
of control and of return of benefits from the program. 

In this paper, the many evaluations of the Botswana 
National Literacy Program (BNLP) are reviewed , with focus on two 
major evaluations one internal, the other external. The 
review will demonstrate that while discussions among the 
various stakeholders were conducted in the language of program 
improvement, the need to intervene and control was part of the 
explanation of when and why the various evaluations were 
commissioned and what use, if any, was made of evaluation results. 

DEVELOPMENT AND PROGRAM CONTEXT 
Botswana { formerly , the British Protectorate of 
Bechuanaland) which became independent in 1966, is in many ways a 
typical Third World country, riddled with poverty, illiteracy, and 
political dependency. It has, however, some special 
characteristics that are noteworthy. It hao a small population 
of around one million, spread over a large country of the size of 
France, two-thirds of which is covered by the inhospitable Kalahari 
desert. It is a country rich in minerals and fortunate in its 
leadership that has decided to use the country's mineral wealth in 
alleviating peoples' poverty. The most important development 
goals of the country are employment creation and rural 
development. Both require sizeable inputs of education and 
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extension for implementation. The government has, first, assigned 
an important role to out-of -school nonformal education; and, 
then, made print communication central to the delivery of 
nonformal education and extension. The policy is rooted in the 
realities of a significant part of the population bypassed by 
the formal education system; the impossibility of being able to 
provide face-to-face extension services to farmers in widely 
scattered villages and cattle-posts for lack of personnel and 
infrastructures; the inadequacy of the radio broadcasting network 
and its i..nerent inability to carry the whole communication 
burden; and the hope that a literate people would become 
independent, effective and frequent users of development 
information (Bhola, 1985). 

Botswana's mineral wealth does not make it self- 
sufficient, by any means. The country needs both technical and 
material assistance to overcome underdevelopment; and it is, 
indeed, receiving considerable foreign technical assistance in 
all the various sectors of development. The BNLP is one of the 
many externally funded project. Inputs from the Government have 
been relatively small. The 1987 evaluation study of BNLP 
(Gaborone, et al., 1987: 92) estimated that within the 
development budget, donors have provided 91% of the development 
expenditure since 1980 and 72% of the total costs. The Swedish 
International Development Authority (SIDA) provided 42.5% while 
36.4% came from the German Agency for Technical Assistance (GTZ). 
SIDA has chosen not to be directly involved in the day-to-day 
working of the BNLP, but GTZ has. During 1982-84, GTZ sent a 
German expert who was appointed Literacy Coordinator to head the 
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BNLP. GTZ has also commissioned a number of evaluations of the 
BNLP. 

The history of the BNLP is a few years longer than its 
association with GTZ. After some pilot testing in 1979 and the 
experimental year of 1980, the BNLP had been launched in June 
1981 to cov( - all the nine districts of the country. All 
illiterate adults and youth ten years and older were targeted by 
the BNLP. T^ ey were estimated to be some 20% of che relevant 
population, between ?50,000 to 300,000 in number. Illiteracy was 
to be completely eradicated by 1985/86, that is, during the life 
of the Fifth National Development Plan (NDP5), covering 15,000 
adult learners in the experimental year of 1980 and then adding 
50,000 a year until all of the illiterate population would have 
been covered. Each cohort of ^.dult learners would recieve 12 
months of guided and supervised instruction in reading and 
writing in Setswana. At the end of 12 months they would be given 
self-instructional materials to be learned independently by each 
learner (Bhola, 1985). The enrolement targets envisaged in NDP5 
have proved to be unrealistic. The Sixth National Development 
Plan (NDP6), 1985-91, has already extended the BNLP, enabling it 
to operate until illiteracy is fully eradicated from the country 
(Botswana Government, 1985). 

CONCERNS WITH EVALUATION 

The BNLP is perhaps one of the most extensively 
evaluated national literacy programs in recent memory (Bhola, 
1984b, 1985; Bhola, et al., 1983; Gaborone et al., 1987; Unesco, 
1976). Starting from 1979 upto the tin:e of writing, it has 



Botswana Literacy Evaluations 



undergone multiple evaluations , both large and small (Gaborone et 
al., 1987). The BNLP, again, may be the only literacy program in 
the Third World with its senior program staff so extensively 
trained in evaluation planning and implementation (Bhola, 1984a). 

A Pre-natal Evaluation, 1978 

The seeds of the BNLP as well as of concerns for 
evaluation were sown in a pilot project of functional literacy 
implemented and evaluated by the Botswana Extension College 
(1978). The pilot project had been developed especially for 
gaining experience for conducting the BNLP to come later. 
In terms of professional knowledge, this fjvaluation produced 
facts and insights that hold true ten years later. 

This evaluation covered about ] ,335 learners in some 59 
learner groups in three pilot areas; and used class records, tests 
and observers as sources of data. The report had found that 
learner recruitment was no problem, there were always more 
learners than could be served. However, 82% of enrollees were 
females and 77% were in 15-30 years age range. Learning to read 
and write one's own letters, received from husbands who were away 
working in the mines in South Africa, was a predominant motive 
for joining classes. Only one in five students dropped out but 
irregular attendance was widespread*. Group leaders dropped out 
about half as frequently as did learners. There was 
dissatisfaction with the amouni of honoraria received by Literacy 
Group Leaders . 

Formal testing of adult learners had been an almost 
complete failure. On the basis of visits it was concluded that 
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in "successful groups" learners had shown good progress in reading 
and writing in 15 weeks, but for most groups 15 weeks was not 
enough. Most learners would have been unable to read unseen 
materials. Radio helped recruit learners but did not teach them 
anything useful. Teaching materials were inadequate, as were 
follow-up materials. Those who learned to read and write did not 
learn to discuss. There was no evidence of transfer of learning 
to action projects; and self-awareness, another objective of the 
functional literacy project, was rarely achieved. Weak leaders 
were found to belong to too many local clubs and organizations. 
Strong leaders followed the teaching method most closely. 
Leaders themselves were unable to initiate discussion within 
groups. It helped the discussion if the group was segregated by 
age and if people were arranged in pairs. Supersivion was found 
to be an important need. The most important lesson offered to 
evaluators was: Always go and look for yourself. Records 
and test data are not enough (Botswana Extension College, 1978). 

Evaluation for Resource Commitment : 1981 

A Department of Nonformal Education (DNFE) had been 
established within the Ministry of Education of the Government of 
Botswana in 1978 and the BNLP aunched in 1980 (Roth, 1987). In 
planning the BNLP, the planners had made an excellent use of the 
experience in literacy work already available in Botswana 
(Botswana Extension College, 1978; Botswana Government, 1979). 
However, before committing assistance to BNLP, GTZ commissioned 
their own evaluation of what was on the ground (Roth and 
Etherington, 1981 ) . 
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The evaluators were impressed with the fact that the BNLP 
had succeeded in establishing a functional infrastructure for 
program delivery and had conducted extensive training of field 
staff — Literacy Assistants (LA's) and Literacy Group Leaders 
(LGL's). Methods and materials were seen as effective overall; 
and the program was viewed as meeting the needs and demands of 
the people. 

On the debit side, the evaluators found insufficient top 
level commitment; little involvement of extension agencies in a 
literacy program that sought to teach functional economic and 
development skills to the participants; and high staff turnover 
at the top and at the middle levels of tha program. To teach 
functional literacy skills, the report suggested two or three 
four-month teaching sessions instead of only one as planned by 
the project. Even on the basis of one four-month session as 
initially planned, the target of 50,000 a year was seen as 
unrealistic and unlikely to be met. The continued paucity of 
follov7-up materials was also noted. 

The evaluation recommended that GTZ should collaborate in 
the BNLP by providing technical assistance and suggested that 
evaluation remain a critical element in a program of this scope. 
The report recommended that a mid-term evaluation of the BNLP 
take place in late 1982, including DNFE staff members and 
experienced external consultants. The Project Memoranrum signed 
later (Botswana Government, 1982), mandated such an evaluation; 
and the Education Planning Unit of the Ministry of Education 
initiated action on it in early 1982 (Kann^ 1982). 
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It is interesting to note that important recommendations 
of the Roth-Etherington evaluation regarding the need for an 
articulation of the political commitment, improved .coordination 
between BNLP and development extension agencies, and production 
of more follow-up reading materials were not systematically pursued. 

THE 198 3 INTERNAL EVALUATION: 
LOCAL OBJECTIVES, LOCAL CONTROL 

The internal evaluation of 1983 received inputs both from 
GTZ and the German Foundation for International Development . 
(DSE). The DSE would train local program officials, particularly 
the District Adult Education Officers (DAEO's) in evaluation 
techniques; ^nd the DAEO's would then become the major 
instruments of implementing the internal evaluation. An 
international evaluation consultant, with experience in training 
and evaluation on Unescc and ea. lier DSE projects, was invited to 
provide the technical leadership in behalf of both GTZ and DSE 
(1). GTZ 's local representative, the Literacy Coordinator of 
BNLP, would oversee the actual implementation of the evaluation 
exercise. The Planning Unit of the Ministry of Education would 
perform further watchdog functions. 

There were no written terms of reference in regard to the 
evaluation exercise, about questions to be answered and the 
methods to be used in answering them. Until after the evaluation 
was almost complete, the evaluation team had got nothing from the 
GTZ but encouragement. The Evaluation Consultant, the DSE, the 
local representative of GTZ, and officials of the Ministry of 
Education were all agreed upon the evaluation approach. 
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The choice of the approach came from earlier evaluation 
experiences in the Third World and from the Zeitgeist of the 
early 1980s. Experimental and quasi-experimental approaches to 
evaluation had been seriously questioned during the 1970s. There 
had been talk of responsive evaluation (Stake, 1975), 
illuminative evaluation (Parlett and Dearden, 1978), evaluation 
as connoisseurship (Eisner, 1979), participatory evaluation 
(Hall, 1978), utilization-focussed evaluation (Patton, 1978), 
evaluation in the naturalistic mode (Cuba and Lincoln, 1981), and 
evaluation as history (Cronbach, 1980). Evaluation practitioners 
h^^d been advised to distinguish between the "context of command" 
and the "context of accommodation" as they cnose evaluation 
methods. A paradigm shift was indeed in the process of taking 
place (Polkinghorne, 1983; Reason and Rowan, 1981). 

The locus of the evaluation team "^n Botswana, it was 
tacitly agreed, would not be on science but on sense-making. The 
normative objectives would not be objectivity, reliability and 
validity, but credibility and coherence. The team would not 
experiment through a stand-alone evaluation (Cronbach, 1980), but 
provide an experience that would bring to practitioners "the 
essential qualitiative knowledge of what happened" ( Campbell, 
1979:71). They would be invited to look at their ov;n program 
with a critical eye and search for causes, not culprits (Bhola, 
1979, 1982). Only such an evaluation would have the best chances 
of being utilized later to make changes in program implementation 
(Struening and Brewer, 1983). Finally, the evaluation will seek 
to project multiple value positions, portraying interests not 
only of donors but of all stakeholders. 

10 
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The methodological approach would be both eclectic and 
dialectical. The evaluation would use both good logic and 
sharpened perceptions; will tackle both feelings and facts; and 
thus be both quantitative and qualitative. Formal instruments 
such as inventories r questionnaires and interview schedules will 
be used but they will be both structured and unstructured; and 
these will have the possibility of becoming excuses for 
conversations with respondents, if field realities so required. 

OPERATIONAL PLANNING OF THE EVALUATION EXERCISE 

The evaluation was conceptualized not only as an internal 
evaluation but also as building a built-in evaluation system for 
providing continued organizational intelligence. The purpose was 
to make the total program system conscious of information needs 
to monitor inputs, processes, outputs and outcomes; and to 
establish a data collection, storage, retrieval and utilization 
system a Management Information System (MIS) — commensurate 
with needs (Davis, 1974). 

The preceding set of objectives required that local 
capacity be built through t raining . The Action Training Model 
already tested-in-use in Kenya was used (Bhola, 1983). The more 
than 15 DAEO 's were targeteu for suc*^. training. These DAEO's 
would be trained in the techniques of evaluation planning, design 
and implementation in a series of short evaluation workshops. 
They would, in turn, provide the necessary orientation to about 
120 Literacy Assistants (LA's) in charge of various clusters in 
each district. The 3,000 or more Literacy Group Leaders (LGL's) 
would also be associated with the conduct of the evaluation 
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exercise. Every one of them later would participate in the 
development of the Management Information System, 

An important decision was made about data flow from the 
field 'he center in Gaborone. - The principle enunciated was: 
Use before Dispatch! One who collects, collates or aggregates 
data must first use it to improve the program at one's own level 
before releasing it to the level above. The LGL 's were, 
therefore, asked to collect and collate data at the level of the 
learning center, in tables especially provided, and learn about 
their centers themselves before sending data up to the LA's at 
the cluster level. The LA's would collate all data at the cluster 
level, in tables specially designed for the purpose, and write 
cluster reports on the basis of available data. They would, to 
the extent of their jurisdiction, act upon what they had learned 
from their reports. Then, hey would send their reports along 
with supporting data to the DAEO's, The DAEO 's would collate all 
data district-wise and write district reports. They too would 
act on their own reports, to the extent of their responsibilities, 
and then send their reports to the HQ in Gaborone, There a 
national reporu will be written and used in decision making at 
that level and in negotiations with other government department? 
and outside funding agencies, 

THE IMPLEMENTATION OF THE EVALUATION EXERCISE 

The dialectical approach continued during the 
implementation phase the dialectic oetween the ideal and the 
possible, between the standard and the satisficing, between 
technical rationality and ^ocial processes was sustained.. The 

12 
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evaluation became unabashedly multi-schematic and multi- 
paradigmatic (Polkinghorne, 1983:xi), synthesizing contradictions 
as they appeared. Norms were projected, but approximations were 
accepted as long as the process kept on moving and participants 
did not get stuck (Schon, 1983). The sole methodological 
objective was to describe, as best as possible in the context, 
the scope and the meaning of the BNLP as it was being 
implemented . 

A rather formal matrix was imposed on the evaluation 
exercise (B^ola, 1982, i984c): the BNLP was analyzed as a system 
with various constituent subsystems; information needs of various . 
subsystems were listed; outlines of a Management Information 
System were hammered out; an evaluation agenda relevant to the 
BNLP at that point in time was agreed upon; evaluation studies 
were designed in terms of the agenda; tools and instruments 
needed for data collection were prepared, translated and where 
possible pre-tested; field work was done; and data obtained was 
collated and computer-analyzed. 

Within this formal matrix, however, there was tolerance for 
the approximation. Evaluation was being conducted, not within 
the context of control and command, but in the context of 
accommodation. The formal matrix had to have elasticity. The 
realities of lack of control, congruence and communication had to 
be faced. 

The training inputs necessary for building a built-in 
evaluation system had to be in English. While these training' 
inputs had been designed with meticulous care, the usefulness of 
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this training varied from one DAEO to another, depending upon 
their mastery of the language. There Wc.s also the constraint of 
training time. Decisions in regard to the information needs of 
the system, evaluation agendas and the elements of an MIS had to 
be made by the DAEO's and accepted even if they seemed less than 
elegant and comprehensive to the outsiders. 

All tools and instruments had to be designed by the 
DAEO's individually or working in small groups. FacuTty 
resources at the training workshops were adequate, yet there was 
a limit to how much quality control could be imposed and what 
kinds of revisions could be made. DAEO's could not always 
trap-,iate the instruments they had constructed, nor did they 
always agree on the translations made by others. Setswana is a 
new subject in the school system, introduced in the post- 
Independv^nce period and good translators are hard to find. 
Checks on the quality of translation by retranslation was not 
always possible and pre-testing was also not always done for lack 
of technical resources and constraints of time. 

The training of LA's as investigators had to be left to 
the DAEO's and was not always done systematically or well. 
Dependable data on the scope of the current program was not 
available for a sampling frame to be devised, and only general 
decisions could be made about who should be interviewed and 
where . 

Da collection could not be supervised in an evaluation 
that was conceptualized as an experience for the whole system. 
There were open-ended questions in each and every instrument. 
Some questions in the questionnaires and instruments did not 
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communicate and the investigators ended up using these 
instruments not as yardsticks, but as frames of reference for 
conducting conversations with their respondents. Data was 
recorded in Setswana and English and in some cases was 
translated from a language other than Setswana. Once available, 
this data had to be coded, again, back to English using themes 
and key words from the English language. 

TRAINING FOR INTERNAL EVALUATION 

The various stages of the process of building a built-in 
evaluation system, involving both training and action, are 
presented below: 

Evaluation Workshop I ( November 15-26, 198 2 ) 

Workshop I was conceived basically as a training 
workshop (2). It had two main objectives: (i) to learn the 
process of evaluation planning by doing evaluation planning; and 
(ii) to write proposals for evaluation studies on topics relevant 
to varioufs districts, complete with objectives, designs, and 
drafts of instruments. 

Evaluation planning was taught using the systems 
approach. The "evaluand," the BNLP, was analyzed as a system 
of overlapping subsystems: (i) policy and planning subsystem ; 
(ii) administrative and instructional delivery subsystem; (iii) 
technical support subsystem; (iv) social mobilization subsystem; 
(v) curriculum and materials development subsystem; ( vi ) teaching 
learning subsystem; ( vii ) post- literacy subsystem; ( viii ) 
training subsystem; and (ix) evaluation subsystem (Bhola, 1984c). 
An ideal-type catalog of information needs of the various 
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subsystems was developed and made the basis for developing a 
situation-specific evaluation agenda for the BNLP. This agenda 
was validated through the social process of developing a 
consensus among the DAEO's and the HQ staff. It could not be 
taken to any other constituency at any other level. 

A distinction was made between the need for an MIS 
wherein program data routinely generated by the program 
could be stored for retrieval and utilization as and when 
appropriate; and evaluation studies conducted specially for 
meeting unanticipated information needs of the program. The new 
administrative patterns and role definitions necessitated by the 
built-in evaluation system were discussed as well. 

In the second phase of Workshop I, participating 
DAEO's were asked to select an information need they had 
already experienced in their work in the district and to design 
an evaluation study that would generate the information required. 
They were assisted in the design of their evaluation studies, in 
selection of indicators, sampling procedures and in the 
preparation of tools and instruments. Plans were made for 
translation and pre-testing of instruments and for field work, 
before the DAEO's leturned to work in their districts. They 
would come back to attend another workshop some 15 weeks later 
where they would bring their data for collation and report 
writing . 

Evaluation Workshop J I (March 7-18, 1983 ) 

The data brought to Workshop II by the returning DAEO's 
was uneven both in quality and quantity. Some of the DAEO's, 
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back in their districts, had been unable to command the 
secretarial and transportation resources needed for the field 
work. Earlier aspirations were lowered in almost all cases, but 
all DAEO's came to Workshop II with some data in hand. 

More importantly, they had a] 1 gone through their 
initiation as evaluators. They had all finalized their 
instruments in some form, had translated these instruments as 
uest as they could and had administered them to samples of 
respondents. 'mey had realized that respondents were not always 
accessible. They were frustrated by the realization that their 
questions were not always understood by their respondents and 
that they sometimes invited answers that were inappropriate or 
ununderstandable. They also understood the need of training 
their LA's if they were to be deployed as investigators. 

These studies were based on rather limited data collected 
from a single district, and sometimes from a single cluster in a 
district, yet as the data were processed in tables or around 
themes, numerous significant insights began to emerge. 

Findings from Training Exercises in Evaluation 

A summary of the ten completed (3) studies brought to 
Workshop III for sharing with other participants should be of 
interest. These Individual Reports told us that the coverage of 
the program was far below the level of expectation as included in 
the NDP5. Common folks saw the program somehow promoting rural 
development even though only the 3R's were being taught and the 
teaching of functional skills was being neglected. 

There was a lack of facilities and materials. Yet people 
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were hungry for knowledge and had high expectations from the 
program. They wanted to learn about farming, health, building 
crafts, small businesses, and some wanted to learn English. In 
one area, 60 per cent of learners wanted to use BNLP to go inro 
further formal education and another 40 per cent wanted to use 
the program as a springboard for enonomic income-generating 
projects. Both these hopes seemed doomed to failure since 
no interfaces had been built between the formal and nonformal 
subsectors of education or between BNLP and the country's 
development extension services. Indeed, some extension workers 
were completely unaware of the existence of the program; and some 
others who knew of it were uncooperative. 

People were indeed learning to read and write. Some would 
not mind learning from a primer that would teach Setswana as a 
second language, but the fact that literacy was taught in 
Setswana from the same one primer to all language groups did not- 
seem to pose any drastic problems. It was perhaps only slowing 
the process of learning and teaching but did not force dropouts. 

Already people were beginning to use what they had learned 
by way of literacy and numeracy. However, the new learners were 
not engaging in any "praxis." Their learning thus far had 
neither increased their participation in democratic institutions 
nor increased their productivity. 

The studies pointed to the need for sizeable amounts of 
post-literacy materials; greater inputs into the training and 
supervision of LGL 's who were under-educated, too young to handle 
social and technical issues included in the literacy primer, 
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often frustrated and almost always uncommitted. Specifically, 
one study discovered that LGL's with higher levels of 
qualifications performed less well (perhaps they were more 
frustrated and looking for an opportunity fo jump ship and move 
to something else) than those with only primary education (for 
whom alternative economic opportunities were perhaps fewer, if at 
all). The studies also discovered important training and 
supervision needs of LA's. 

These district reports identified the following urgent 
needs of the program: extension of coverage and increases in 
materials production; close cooperation with and from other 
extension services; and additional training, and better 
incentives to LGL's. They suggested also that more investigation 
of the following topics should be undertaken: Methods and content 
of training functionaries and of teaching adult learne]*s; 
possibility of developing a network of institutions and persons 
wlio could be deployed for follow-up in teaching functional skills 
to learners; and improvement of management and decision-making 
for and within the BNLP. 

Most of the findings fell within the range of expectations 
of the DAEO's and others working on the program and thus 
reinforced already existing knowledge. Some findings, however, 
went against conventional wisdom and were hotly contested; and 
the studies that had produced those findings were challenged by 
workshop participants on the basis of methodology used, selection 
of samples and design of instruments. It is clear from the 
perspective of early 1988 and with the benefit of hindsight that 
all the later findings from the many different evaluations may 
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not have added significantly to what these ten little, 
impressionistic, unscientific studies had made available. The 
echoes of the 1979 pilot project evaluation are also unmistakable. 

Evaluation Workshop m (May 9-20, 1983) 

Workshop III was designed to provide the transition from 
a training phase to implementation phase of the national built-in 
evaluation system. The workshop was focussed on the core 
objectives of (i) recollecting the list of information needs of 
BNLP developed at Workshop I; (ii) prioritizing those information 
needs as a group; (iii) developing agreement on data to constitute 
an MIS and on the formats for such data collection and storage; 
(iv) listing urgent information needs that must be fulfilled 
through specially designed evaluation studies; and (v) designing 
such evaluation studies, complete with tools and instruments for 
implementation at the national level. It was nothing short of 
amazing that all these tasks were completed at a satisf icing 
level within the auration of the workshop. 

A Visit to the Field to Prepare for Implementation 
of the Evaluation in the Field (August 15-26, 1983 ) 

To ensure that things were happening as planned, the 
author returned to Botswana for a field visit some 12 weeks after 
Workshop III. All was not well on the ground! Getting satisfactory 
Setswana translations of tools and instruments had not been easy. 
Plans for the pre-testing of instruments fell through when the 
University of Botswana lecturer who had agreed to supervise the 
pre-testing left the country. The printing of instruments — 
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involving close to a 100,000 sheets of paper .in all had 
strained the system to the limit. These instrumenrs were waiting 
to be delivered to the various districts by truck which would 
take the driver more than two weeks on the road to make the full 
round . 

Down at the level of districts, things were no better. 
The DAEO's had not always provided the necessary orientation to 
their LA's to prepare them for their role in the evaluation 
exercise. They had sometimes told them what to do, but- not how 
to do it. Detailed schedules for field visits were no where to 
be found. 

Indeed , the LA 's were often anxious and sometimes even 
hostile to the idea of the evaluation exercise. Soire of them 
suggested that what the evaluation exercise was trying to find 
was already known; and needed was implementation and not more 
evaluation. Others said that there was no scope for improving 
learner participation in the program in the midst of a drought 
when people were starving and had left their villages in search 
of food. 

In response to the existing situation, the field visit got 
transformed into a series of 1-day workshops for LA's and th^xr 
DAEO's in each of their district headquarters in the country, 
excepting two which were covered later. Using the data recently 
available from the national population census, the LA's were 
shown distributions of illiterates in their clusters and 
encouraged to recruit them into the BNLP and to serve their needs. 
The instruments to be used in the evaluation exercise were 
distributed to the LA's. The intentions of the various items in 
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those instruments were discussed and some basic procedures of 
data collection were presented. The sampling methods to be used 
in data collection were also discussed. 

At the end of the field visit, it was not clear how the 
DAEO's and LA 's will respond to the challenge of e^^aluation then 
before them. The situation seemed hopeful and hopeless by turns. 
To introduce some certainty into the situation a "fail--safe" 
strategy was devised. The HQ staff in Gaborone, under the 
leadership of the Literacy Coordinator, would conduct an 
evaluation-within-an-evaluation. As the national evaluation 
exercise was going on,' they would work with a small national 
sample controlled from the HQ in Gaborone. It would be assured 
that data on this small sample would be collected and become 
available for analysis. If all else failed, they would have at 
least a stand-alone evaluation to fall back upon. 

The Work in the Field, Impressive by any Means. 

Following the field visit by the author, the Literacy 
Coorainator and his colleague at the KQ in Gaborone were out in 
the field most of the time during the data collection phase, 
going from district to district and from cluster to cluster. 
This may have turned thing*^ around. The field visit earlier 
during August 15-26, 1983 may have shaken things up a bit as 
wel 1 . 

Whatever the causes, the consecyuences were most 
gratifying. During the 10 to 12 weeks period before mid-December 
1983, 17,000 instruments had been completed and delivered: 7,000 
dealing with learning needs of participants; 2,000 dealing with 
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the special needs of non-Sei:swana speakers; 1 , 600 on motivations 
of regular attenders; 1,150 dealing with those who had dropped 
out of the program ; 3,000 listing uses to which literacy had 
been put by the new xiterates; and 1,500 village profiles, 
describing what resources were available at the community level 
that could be used in providing services to the rural people. 

Samples were in flux. Various items in the evaluation 
instruments were found ambiguous and, therefore, unusable. The 
instruments in the hands of many LGL 's and LA's became frames of 
reference for conversations with respondents rather than 
instruments in the classical sense. Yet, there was lot of 
counting for the delineation of the scope of the program and lot 
of questioning in search of the meanings of the program. The 
evaluation exercise also became an instrument of mobilization for 
recruiting participants into the BNLP. 

Seeing the big surge of data rolling in, the HQ in 
Gaborone did not work on the fail-safe plan of an evaluation- 
within-an-evaluation, but decided to buy a microcomputer for 
electronic data processing. To beat deadlines for report writing 
imposed by higher level decision makers at the GTZ headquarters, 
data v;as commandeered from the DAEO's and sometimes directly from 
la's, before they had had the chance to process data at their 
levels and to write their reports as originally planned. Thus, 
the initial plans for data flov? and information utilization were 
abandoned. The conceptualization of evaluation as experience was 
compromised somewhat because the product of evaluation (the 
report) was separated from the process of evaluation. 
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Evaluation Workshop IV ( December 11-17, 1983 ) 

The December 1983 workshop had initially been planned as a 
workshop for finalizing district level reports and for sharing 
the findings. The DAEO's were to write their reports using the 
same one standardized format to enable comparison and cumulation 
of results for the purposes of a national report. Now that there 
was to be electronic processing of data, and most data was 
already at the HQ in the process of being coded, the program of 
the workshop had to be changed to fit new realities. The DAEO's 
spent their time at the workshop developing "dummies" for their 
district reports, completing portions dealing with the 
ecological, economic, social and developmental context of their 
districts as well as in recording the human aspect of the 
evaluation exercise as they experienced it. 

The Aftermat h to Data Collection . An outside technician 
was brought in from abroad to help in the electronic data 
processing. Data was coded with the help of students from the 
Universi^/ of Botswana. It may be recalled, that che evaluation 
instruments had not initially been designed for computer 
analysis. Coding schemes had to be imposed on the data after the 
fact. This was not a simple task. The task was further 
complicated by the fact that the foreign technician was -denied 
direct access to data because of the language barrier. He had to 
depend on local coders and judge the quality of data indirectly 
from the questions these coders happened to raise. A report was 
written in 1984, a bit later than expected ( Hundsdorf er , 1984). 
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AN EVALUATION OF THE INTERNAL EVALUATION 

The consensus about the usefulness of the evaluation broke 
down even as data were being processed. The donor had perhaps 
expected an evaluation controlled by their resident expert and 
consultant. GTZ may have thought that local DAEO's would 
be trained as field in\ estigators not as equal participants in 
the evaluation process. The attack from the GTZ office in 
Germany was essentially methodological. The donor found the 
participatory methodology to compromise objectivity. The 
omission of literacy and numeracy tests was found i itolerable; 
and data actually collected was considered sub-standard. 

Interestingly enough, everyone in Botswana directly or 
indirectly involved with the evaluation exercise was happy with 
the evaluation approach as well as with the results. There were 
several reasons tor satisfaction. 

As already indicated, some 17,000 questionnaires and 
interview schedules were completed as part of the evaluation 
exercise. The evaluation counted 28,800 learners enrolled in the 
program, ranging in age from 6 to 94, 31% males and 69% females. 
Some 65,000 learners had been covered during the life of the 
program which had graduated some 20,000 literates. 

The DAEO 's and LA's interviewed in the course of May 
1984 writers' workshop (4) were most gratified with the 
evaluation training they had received which had made them better 
pr o 1 63 sionals all around. 

All associated with the evaluation exercise had found the 
experience meaningful. They were glad for the opportunity to 
have been in the field, and to have interacted with the people 
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whose lives they had sought to change. Here and there ^ they had 
seen at first hand the consequences of their program actions. 

The evaluation would have to be considered successful in 
regard to the most important criteria of all: The resulting 
understanding of functionaries of the BNLP showed much 
"improvement over previous understanding" ( Polkinghorne , 1983:3). 
Both tli€ scope and meaning of the program were clearly 
understood. They had discovered u'^-at motivations were not 
spontaneous and that mobilization was necessary for recruitment. 
They had developed a real feeling for the disadvantages from which 
the illiterate suffered even in "th back of the beyond" in the 
Kalahari desert. They had understood how literacy was indeed 
changing the patterns of leadership in local communities. 

They had discovered that some of those they had been labeling 
as dropouts were really completers. These learners had left 
because the learning centers did not offer them much any more. On 
the other hand^ some learners did not want to leave. The 
learning centers were fulfilling their special social needs. It 
was discovered that literacy cl jses in Botswana do not just last 
12 weeks during the months of September to December but that some 
of these continue the year round . 

The evaluation also discovered primary schools within 
literacy classes. In remote areas of Botswana where primary 
education had not reached^ the literacy class had acquired a dual 
meaning: it was both a literacy class and a primary school. 

In spite of all the f rust rat ions of data col lection ^ 
participants in the evaluation were quite satisfied with the 
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methodology used in the evaluation and with the findings. 
Neither their own experience, nor commonsense would challenge 
what they had discovered. The data compelled, but did not 
surprise. If the data could not be put to inferential uses, they 
certainly could be put to a multiplicity of suggestive uses in- 
developing and modifying the program both in the long and the 
short run. While there may have been a few discordant notes here 
and there, the overall coherence of the data was remarkable. 
It fitted with reality. It made a lot of sense. Some things 
could be done with it (Cronbach, 1980; Cuba and Lincoln, 1981). 
It not only met the truth test, it also seemed to meet the 
utility test (Weiss and Bucuvalas, 198C). 

The participants in the evaluation process found the 
results usable even before reports had been written. Since the 
data and these many insighr had personalized meanings for the 
LGL 's , la's and DAEC's, the evaluation results were utilized in 
making program decisions even before any formal report came out. 

The incidental returns on the evaluation exercise were 
many. First and foremost, the evaluation exercise led to an 
impressive social mobilization in behalf of the program. In many 
cases, the evaluation provided the first opportunity for DAEO's 
and la's for a personal encounter with learners in the villages 
and cattle-posts. As a result, participation went from 18,000 to 
some 28,800. Institutional mobilization also resulted. The BNLP 
was able to establish networks with other government departments 
of education and extension for providing integrated services to 
adult learners at the field level. 

There was further useful fallout! Planning and evaluation 
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are two sides of the samo coin. Evaluation planning is the other 
side of program planning. In learning to describe the literacy 
program as a system in design terms for the purposes of 
evaluation, the DAEO's developed useful insights in the planning 
process per se. The DAEO's, after the evaluation exercise, 
were writing their routine monthly and quarterly reports with new 
insights. The reports were no longer merely descriptive but had 
become quite analytical. For the first time analyses at the 
learning group level and the cluster level had appeared in these 
periodical reports . 

The morale of those working within the BNLP improved. Many 
LGL's, la's and most DAEO 's came to own the evaluation exercise 
and to make references to it both in conversations and writing. 
There was an awareness of the need for professionalization 
in adult literacy planning and some LA 's later joined diploma 
and certificate courses in nonformal education and adult literacy 
under the department of correspondence education of the 
University of Botswana. 

There was some fallout on other department of the Ministry 
of Education. The Correspondence Education Department of the 
Ministry was now anxious to train their officers in evaluation in • 
the same mode preparing them to apply those bkills to course 
evaluation and learner evaluation. There was also interest in 
curriculum evaluation and one of the later workshops for 
teachers engaged in curriculum evaluation used the project 
handbook, Evaluating Development Training Programs as text 
(Bhola, 1982). 
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People at the headquarters and in the Ministry were also 
satisfied. The Government of Botswana, Ministry of E'duration 
newsletter, Thuto (1983), wrote: "We ought perhaps to mention 
that the evaluation has been a very valuable source of 
information. As a result, we have much more information about 
the workings of the programme and if it becomes necessary to 
change it in certain ways, we can confidently take decisions that 

are in line with the findings of the evaluation The programme 

is already being modified as suggested by the evaluation." 

Even though, the e\aluation was rejected by the donors who 
accused their own resident representative of not having put it 
into the policy making system, there is evidence of use of the 
evaluation by the local decision makers. Many incremental 
improvements were made at the operational level- The NDP6 document 
acknowledges its debt to the evaluation when it says: "Following 
the experiences of the first years of the programme and 
recommendations made in the internal evaluation of 1983, a new 
strategy for the long-term eradication of illiteracy has been 
worked out." (Botswana Government, 1985: 158). 

THE 1987 EXTERNAL EVALUATION: 
TO SERVE POLICY PURPOSES 
As the end of GTZ grant period approached, GTZ was 
considering withdrawal from the BNLP. To make an "objective 
decision", GTZ ordered an evaluation. This will have to be 
an external evaluation, to be objective; and methodologically 
sound, to be dependable in decision-making. Early in 1986, the 
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Institute of Adult Education (" .i5 ) of the University of Botswana 
was commissioned to conduct such an evaluation. 

The evaluation document describes itself as an external 
evaluation, but it falls somewhat between the internal and the 
external. The evaluators included a native Batswana 
professional educator, a Tanzanian teaching ^.t the University of 
Botswana and an expatriate from England who has spent many years 
in Botswana and is most familiar with the Botswana scene. The 
evaluators can be considered to be outside the institutional 
setting of the BNLP but they are not too far away from the 
Ministry of Education. In any case, the three evaluators worked 
in close collaboration of a Reference Group composed cf various 
officials of the Ministry of Education and other officials of the 
Government of Botswana which provided advice, information and 
expertise during the process of information collection. The 
findings and recommendations were developed by the three 
evaluators independently of the Reference Group and is solely 
their responsibi lity . 

The 1987 evaluation report (Gaborone, et al., 1987a), in 
reviewing earlier evaluations, characterized the 1983 internal 
evaluation as "an impressive attempt to utilize program staff to 
evaluate their own work and produce feedback of results which 
could lead to imiuediate operational changes .... There is 
considerable evidence that the evaluation did lead to incremental 
changes within the BNLP's operation," 

The 1983 internal evaluation was criticized, however, for 
never submitting clear cut cone'' ions and never properly reporting 
findings to the National Literacy Committee, thereby, formally 
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putting results into the policy-making channels of the government. 
The new 1987 evaluation report took note of the fact that GTZ had 
criticized the 1983 evaluation on methodological grounds and 
had disliked the participatory approach it had followed. GTZ had 
found ±t statistically weak and lacking in statistics on learner 
achievement obtained through testing. The MIS, the 1983 
evaluation had worked on, was never institutionalized. Some of 
the data on computer tapes — this writer learned for the first 
time — had been lost during shipping back and forth between 
Germany and Botswana and, therefore, the promised detailed 
analyses of data had never been possible. 

THE METHODOLOGY OF THE EXTERNAL EVALUATION 

The methodology of the new evaluation involved a variety of 
data gathering techniques including documentary analysis; field 
observation of literacy classes in progress and attendance at 
training workshops for LA's and LGL's; and individual as well as 
group interviews, both structured and unstructured with DAEO's, 
la's, LGL's and nine key informants in policy making positions. 
Questionnaires were used for a variety data collection 
purposes: an "interview questionnaire" in Setswana administered 
by la's to 845 literacy participants; a self administered 
questionnaire in English to LGL's (out of 1221 such 
questionnaires sent, 741 were returned); a self administered 
questionnaire in English to LA's (out of 131 such questionnaires 
sent, 73 were returned); and a self administered questionnaires in 
English administered to DAEO's (out of 20 such questionnaires 
sent, 14 were returned). 
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The evaluation included a literacy and numeracy test 
administered by LA's to 845 literacy participants. In-depth 
case studies were developed of two income-generating projects. 
All relevant statistics that already existed were analysed. 
Some photographic evidence was also used. 

The sampling frame for learner testing and interviewing 
was developed in consultation with the Central Statistical Office 
and reflected three categories of learners-settings: urban areas, 
minority language rural areas and Setswana speaking rural areas. 
A random proportionate sample generated a list of 83 LGL 's 
and 841 learners. Data analysis was handled by computer 
and a large number of tables so generated have been made part of 
the report. 

RESULTS OF THE EXTERNAL EVALUATION 

The evaluators could not find any reliable data on 
literacy rates in the country or demographic data on target group 
for the BNLP by way of developing a context for their own 
evaluation. Within the BNLP there was no data available to 
distinguish new learners from old ones or to indicate the length 
of stay within the BNLP for various learners. 

By the end of 1986, the BNLP had had a cumulative 
enrolment of 178,000 — a substantial achievement by any 
standards. The BNLP had been the "largest sustaineed adult 
education program since independence." 

Literacy tests revealed a high level of performance: 81 
per cent of those tested achieved a score of 40% or more which 
can be equated with passing the Standard Four attainment test. 
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How long it takes a persor to learn to read and write was net 
possible CO determine, however. Non-Setswana speakers were not 
negatively affected as literacy learners. 

The evaluation indicated that distribution problems within 
the BNLP were significant; teaching content and methods both 
needed review; and learner expectations from the program were 
further education and training for employment both of which 
needed to be better reflected in the program. 

There v/as the need for a more comprehensive policy on 
post-literacy. The BNLP needed to build interfaces with multiple 
points of entry between BNLP and formal education envisaged in 
the NDP6. There was urgent need for developing a special program 
that would be equivalent to the Primary School Learning 
Examination. The needs for learning English expressed by many 
also needed to be met. 

The interface between BNLP and development extension also 
envisaged in the NDP6 needed to be made operational. The DNFE 
should play the mobilizational and organizational role but then 
the relevant extension departments should take over the task. 
The DNFE/BNLP should produce sufficient amounts of follow-up 
reading materials, promote rural newspapers and establish 
village libraries . 

In terms of administration and management of the program, 
the evaluation proposed gre?=iter interface between the National 
Literacy Committee (NLC) and the Rural Extension Coordination 
Committee (RECC) at the center; and between NLC and District 
Extension Committees (DET's) and Village Extension Committees 
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(vet's) in the field areas. A new organizational structure 
was proposed for DNFE itself that would clarify headquarter and 
field staff relationships and permit greater operational 
decentralization. The needs of staff training and schemes of 
service were also clarified. It was recominended that BNLP pay 
special attention to the installation of a "revised" MIS for use 
in day-to-day decision-making. 

Finally, to demonstrate national commitment to literacy 
and to claim ownership of BNLP, the government should increase 
its funding commitment. The external evaluation also recommended 
internal evaluations of instructional materials and staff 
training programs . 

A Self-Evaluation of the External Evaluatio n of 1987 

In an honest and incisive self-evaluation of the 
external evaluation of 1987, the authors of the report write: "It 
should be noted that many of the problems identified (such as the 
need for DNFE to have a research and evaluation capacity and the 
inadequate post-literacy activities) have been referred to in 
many earlier documents on the BNLP. Similarly, many of our 
recommentations are not new b-t echo previous suggestions. We 
therefore conclude that, in the final analysis, the significance 
of an evaluation report such as this lies less in the validity 
of its findings and the strength of its arguments than in its 
timliness and its insertion into the policy making process." 
(Gaborone, et al., 1987a:5). 

The report seems to have served the policy making 
objectives of both GTZ and the Government of Botswana. GTZ has 



34 

tr 

or) 



Botswana Literacy Evaluations 

since agreed to continue support of BNLP, but it may have 
used "objective data" from the "external evaluation" to exact 
some promises from the Government of Botswana in regard to 
funding and day-to-day operations. Already GTZ has made two 
inputs of its own at the operational level as evidenced by 
proposals for a reorganization of the DNFE (Beyer, 1987) and the 
introduction of more logical methods of planning literacy actions 
(Roth, 1987). 

AN OVERVIEW OF EVALUATIONS OF BNLP 

What are the lessons to be learned from an overview of the 
evaluations of BNLP? 

1. Evaluation, especially internal evaluation, is an 
important educational experience insofar as it can make the 
calculus of means and ends of a program transparent for program 
implementers . Internal evaluating should indeed be a continuous 
part of all program planning and implementation. However, there 
may come a point when preoccupation witn evaluation may absorb 
resources needed for implementation and when evaluation may end 
up being no mo:e than a strategy of postponement. Clearly, this 
should not be allowed to happen. At this point ir BNLP, 

it is time to move vigorously towards implementation. 

2. Evaluation will always be a mix of the professional (the 
desire to improve) and the political (the desire to control means 
and ends). Evaluation will never be fully exorcised of politics, 
nor perhaps, it should be. However, the political-distributive 
implications of evaluation should be clarified and the worst 
encroachments of politics resisted through discussion at the front 
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end. Consensus about appropriate terms of reference and about 
methods and strategies of implementation should be attempted, 
even though a breakdown of consensus through changes in 
characters and historical time can never be foreseen. 

3. Special attention should be paid to linking evaluation 
results with the policy making processes. This means that 
timliness and communicabi lity of the evaluation findings are 
of crucial importance. It is also important to insert the 
evaluation findings formally into policy making structures. 

4. Internal evaluations seem to serve better the interests of 
program planners and the objectives of program improvement. 
Internal evaluations seek to keep program control with those 
responsible for program implementation and sometimes with the 
beneficiaries of programs. External evaluations seem to serve 
better the interests of policy makers and the objectives of 
resource allocations between and among various programs. The 
basic intent of external evaluations is to objectify decisions 
regarding reallocation of resources, though the language of 
justification is always of professionalism and progam improvement. 

5. Internal evaluations have greater affinity and tolerance 
for naturalistic methods. On the other hand, external 
evaluations typically prefer the so-called rationalistic methods, 
to create an aura of "objectivity and reasonableness" for the 
reallocative funding decisions to follow. 

6. The experience with internal evaluation in Botswana 
demonstrated the usefulness of the Action Training Model — 
encompassing a series of workshops to provide training to program 
officials in evaluation techniques as they took part in 
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conducting evaluations. It also proved the tenability of training 
"barefoot evaluators" in non-academic settings in the Third World. 
7. Tne evaluation overview also seems to suggest the need for 

the triangulation of evaluation approaches within an overall 
evaluation plan: Management Information Systems to monitor 
programs on a continuous basis; naturalistic evaluations 
conducted internally by program officials to discover the 
meanings of their actions; and more formalistic evaluations to 
make comparisons in those few cases that fit into the context of 
command . 
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NOTES 

1. Contributions are acknowledged trom Dr. Josef Mulier of 
DSE; Dr. Volkhard Hundsdorfer^ GTZ-representative and Literacy 
Coordinator of BNLP; Dr. UKa Kann, SlDA-sponsored Planning 
Officer in the Ministry of Education; and various officials of 
the Government of Botswana in the Ministry of Education. 
Acknowledgements are also made to Dr. Lars Mahlck, International 
Institute for Educational Planning, Paris; Dr. Ash Hartwell, 
Unesco Education Planning Adviser, Maseru, Lesotho; Dr. Peter 
Higgs, Unesco Specialist in Literacy, Lilongwe, Malawi; Ms. Elvyn 
J. Dube of the Institute of Adult Education, University of 
Botswana, Gaborone; and Dr. Linda Ziegahn, USAID Researcher, 
Lesotho Distance Teaching Center, Maseru, Lesotho. Dr. Volkhard 
Hundsdorfer as Literact Coordinator in the Department of 
Nonformal Education, Ministry of Education was responsible for 
all local arrangements in connection with the series of training 
workshops; and was the officer in charge of the implementation of 
the evaluation in all its aspects. Dr. Rainer Hampel, a GTZ 
consultant from West Germany helped with electronic processing of 
data during the latest phase of the evaluation. 

2. Evaluation Workshop I (November 15-26, 1982) was organized 
in collaboration with the International Institute for Educational 
Planning to support the Institute's project: Research and Training 
Project on Evaluation and Monitoring of Educational Reform 
Programs in Africa. Two teams of four evaluators each came to join 
the workshop from Lesotho and Malawi. The March 1983 workshop 

was also attended by teams from Lesotho and Malawi. The May 1983 
workshop received a team from Malawi but not from Lesotho. 
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3. 



This summary was prepared by Dr. Volkhard Hundsdorfer, and 



included these studies: Attendance patterns of learners and 
reasons for absences and dropouts , by T. Chebani; Learning needs of 
adult learners foll^Dwing different occupations, by Mothuseng 
Mathwanye; Learning needs of the newly-literate as defined by 
different stakeholders, by Mothusi George; Administrative 
problems in the delivery of the program as experienced by 
different functionaries, b^. Z. Kwapa; Problems of learners with 
mother tongue other than the language of literacy, that is, 
Setswana, by Reginald Mathangwane; Differences in the performance 
of LGL's with different educational bacKgrounds and attitudes, by 
Ezra Mogw3; An audience survey of listeners of radio programs 
broadcast by the BNLP, by Stephen Mojela; How is the BNLP 
affecting the "small" man in Botswana, by Nnagolo Tau; Are LA s 
retaining and using the social and technical skills taught to 
them as part of their training courses?, by Ted Thebenala; and 
How are literacy programs affected by the total cultural ecology 
of communities where it is offered?, by Dwight Thipe. 
4. The author was in Botswana, again, during August 13-24, 

1984; and had the opportunity of administering questionnaires 
dealing with the usefulness of the evaluation exercise and the 
effectiveness of training for the evaluation exercise. While the 
sample was small — about half of the DAEO 's in the country and 
no more than half a dozen LA's out of a possible 125 there 
were some useful insights that became available. 
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